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USER AUGMENTATION OF CONTENT 



FIELD OF THE INVENTION 

[0001] The invention relates generally to multimedia content and in particular to the 
augmenting of such content by a user. 

COPYRIGHT NOTICE/PERMISSION 

[0002] A portion of the disclosure of this patent document contains material which is 
subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or the patent disclosure as it appears in the 
Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights 
whatsoever. The following notice applies to the software and data as described below and in 
the drawings hereto: Copjrright © 2000, Sony Electronics, Inc., All Rights Reserved. 

BACKGROUND 

[0003] The viewing of multimedia content by consumers is generally a passive 
experience. A consumer is presented with the content, watches it, but typically has no way of 
adding information to the content and sharing the added information with others. In a rich 
multimedia entertainment environment, a consumer may want to add personalized 
information (for example, his critiques of the content, his own list of most interesting scenes 
of a content, etc.) to make their viewing a more active experience. In a home networking 
environment, the consumer may want to share the added information with other members of 
the household. In a wide area networking environment, e.g., the Internet, a consumer may 
want to share the added information with people who are interested in the same content. 
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[0004] Many consumers also desire the ability to assign "bookmark" a particular part of a 
content they are viewing so that they may later conveniently retrieve the content. In a rich 
multimedia environment, a consumer also may want to access other sources related to the 
bookmarked content, e.g. where the bookmarked content is a screenshot of an actress, a 
related source could be another bookmark to a movie with the same actress. Currently, the 
bookmark retrieves only the bookmarked content and the consumer must manually search for 
related information and manually manage the search results. 

SUMMARY OF THE INVENTION 

[0005] Multimedia content presented to a user may be augmented by the user. The 
content may be annotated with information received from the user. Additionally or 
alternatively, the multimedia content may be bookmarked by the user and sources with 
related information are automatically found and associated with the content. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0006] The present invention is illustrated by way of example and not limitation in the 
figures of the accompanying ckawings, in which like references indicate similar elements and 
in which: 

[0007] Figure 1 illustrates an system overview of one embodiment of the present 
invention. 

[0008] Figure 2 illustrates one embodiment of logical blocks for an augmentation 
manager of Figure 1. 

[0009] Figure 3 is a diagram of a computer environment suitable for practicing the 
invention. 
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[0010] Figures 4A-C are flow diagrams of methods performed by the augmentation 
manager of Figure 2. 

DETAILED DESCRIPTION 

[0011] In the following detailed description of embodiments of the invention, reference is 
made to the accompanying drawings in which like references indicate similar elements, and 
in which is shown by way of illustration specific embodiments in which the invention may be 
practiced. These embodiments are described in sufficient detail to enable those skilled in the 
art to practice the invention, and it is to be understood that other embodiments may be 
utilized and that logical, mechanical, electrical, functional and other changes may be made 
without departing from the scope of the present invention. The following detailed description 
is, therefore, not to be taken in a limiting sense, and the scope of the present invention is 
defined only by the appended claims. 

[0012] Figure 1 shows a system overview of one embodiment of the present invention. 
The system 100 includes a display 130, such as a television monitor, an augmentation 
manager 140, a set top box 150, and content sources 160 accessible to the set top box 150. 
Content sources 160 may include but are not limited to nodes on a local area network, the 
Internet, sources of profile information and on-site media system devices. While the 
augmentation manager 140 is illustrated as a component of the set top box 150, it will be 
appreciated that it also may be integrated into the display 130, or connected to the set top box 
150 or display 130 as a separate component. Content sources 160 may be accessed by the 
augmentation manager 140 using any conventional means (e.g. optical, wired, wireless 
transmission) through which a communication connection may be made. The augmentation 
manager 140 associates annotations, bookmarks and related sources, or any combination as 
augmentations to selected content. 
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[0013] The augmentation manager 140 allows consumers to enter an annotation to 
content being presented to them. The annotation is associated with a particular content 
selection and may contain any information the consumer chooses to enter (e.g., his critique of 
the content, the list of his favorite scenes, a related URL link regarding the content, etc.). The 
annotation is added to a list of existing annotations for that particular content. The 
augmentation manager 140 may optionally export the annotated content for immediate or 
subsequent use by an external application (e.g., posting the annotated content in an area 
accessible by others with similar interests). 

[0014] The augmentation manager 140 also allows a consumer to assign a bookmark to 
content and automatically searches for related information from sources determined by the 
consumer's user profile (e.g., specific areas of interest), network connectivity (e.g., local or 
wide area network connections), and the availability of devices associated with an on-site 
media system (e.g.. Media Suponjii™ enabled devices from Sony Electronics). A list of any 
sources with related information found by the augmentation manager 140 is logically 
attached to the bookmarked content. Thus, when the consumer retrieves the content using its 
bookmark name, he may also navigate through the list of related sources and switch to view 
any of the related information. It will be appreciated that the bookmarked content and list of 
related sources may also be exported in a manner similar to the exporting of annotated 
content. 

[0015] Figure 2 shows exemplary logical blocks for the augmentation manager 140 
according to one embodiment of the invention, including presentation module 210, 
augmentation module 220, augmentation retrieval module 230, an optional internal 
augmented content database 240, and augmentation export module 250. It will be 
appreciated that any non-volatile storage device accessible by the augmentation manager 140 
may substitute for augmented content database 240. 
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[0016] The augmentation module 220 facilitates the user augmentation of multimedia 
content, which may be stored in the internal augmented content database 240 or other data 
store for later presentation by presentation module 210. The augmentation module 220 may 
annotate the content with user provided comments and information received through a 
graphical interface presented to the user by the presentation module 210. Existing 
annotations retrieved by the augmentation retrieval module 230 may be edited using the 
graphical interface provided by the presentation module 210, or exported using augmentation 
export module 250 to an external storage device, or to an external application module. The 
augmentation export module 250 may also post the augmentation content to an external 
network for access by others. For example, the presentation module 210 may display the Hst 
of annotations in a scrollable window so that the user can use the window's scroll bar to 
visually navigate the list. When the user finds the annotation he wants to edit or export, he 
selects the annotation, e.g. by clicking on the annotation using a remote control, and chooses 
an action to perform from a menu. 

[0017] The augmentation module 220 also bookmarks user selected content for 
subsequent retrieval by the augmentation retrieval module 230, searches for related 
information, and stores the bookmark and a list of sources for the related information in the 
internal augmented content database 240 or other data store for later presentation by 
presentation module 210. When the user retrieves the bookmarked content, he can view the 
list of related sources and select one or more of the related sources. For example, the 
presentation module 210 may display the list of related sources in a scrollable window. 
When the user sees something on the list of interest, the user can switch to the related 
information by selecting the related source. 

[0018] Figure 3 illustrates a computer environment in which the present invention may be 
practiced. The augmentation manager 140 may execute on a computer system, such as 
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computer system 40. The computer system 40, includes a processor 50, memory 55 and 
input/output capability 60 coupled to a system bus 65. The memory 55 is configured to store 
instructions which> when executed by the processor 50, perform the methods described 
herein. The memory 55 may also store content and augmentations. Input/output 60 provides 
for the delivery and display of the content and augmentations or portions or representations 
thereof. Input/output 60 also encompasses various types of computer-readable media, 
including any type of storage device that is accessible by the processor 50. One of skill in the 
art will immediately recognize that the term "computer-readable medium/media" further 
encompasses a carrier wave that encodes a data signal. It will also be appreciated that the 
computer system 40 is controlled by operating system software executing in memory 55. 
Input/output and related media 60 store the computer-executable instructions for the 
operating system and methods of the present invention as well as the content and 
augmentations. Input/output 60 may also include a network interface to enable the computer 
system 40 to connect to local and/or wide area networks. 

[0019] The description of Figure 3 is intended to provide an overview of computer 
hardware and other operating components suitable for implementing the invention, but is not 
intended to limit the applicable environments. It will be appreciated that the computer system 
40 is one example of many possible computer systems which have different architectures. A 
typical computer system will usually include at least a processor, memory, and a bus coupling 
the memory to the processor. One of skill in the art will immediately appreciate that the 
invention can be practiced with other computer system configurations, including 
multiprocessor systems, minicomputers, mainframe computers, and the like. The invention 
can also be practiced in distributed computing environments where tasks are performed by 
remote processing devices that are linked through a communications network. 
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[0020] Turning now to Figures 4A-C, the particular methods of the invention are 
described in terms of computer software with reference to a series of flow diagrams. The 
methods constitute computer programs made up of computer-executable instructions 
illustrated as blocks (acts) 401 until 409, 411 until 421, and 431 until 445 in Figures 4A, 4B 
and 4C, respectively. Describing the methods by reference to a flow diagram enables one 
skilled in the art to develop such programs including such instructions to carry out the 
methods on suitably configured computers (the processor of the computer executing the 
instructions from computer-readable media, including memory). The computer-executable 
instructions may be written in a computer programming language or may be embodied in 
firmware logic. If written in a programming language conforming to a recognized standard, 
such instructions can be executed on a variety of hardware platforms and for interface to a 
variety of operating systems. In addition, the present invention is not described with 
reference to any particular programming language. It will be appreciated that a variety of 
programming languages may be used to implement the teachings of the invention as 
described herein. Furthermore, it is common in the art to speak of software, in one form or 
another (e.g., program, procedure, process, application, module, logic...), as taking an action 
or causing a result* Such expressions are merely a shorthand way of saying that execution of 
the software by a computer causes the processor of the computer to perform an action or a 
produce a result. It will be appreciated that more or fewer processes may be incorporated into 
the methods illustrated in Figures 4A-C without departing from the scope of the invention 
and that no particular order is implied by the arrangement of blocks shown and described 
herein. 

[0021] Referring first to Figure 4A, the acts to be performed by a computer executing one 
embodiment of an augmentation manager method 400 are shown. The method 400 receives a 
content selection (block 401) and input (block 403) from the user. If the user input specifies 
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it is annotation information (block 405), the method 400 performs an annotation method 
(block 407) described next in conjunction with Figure 4B. Otherwise the method 400 
performs a bookmark method (block 409) described further below in conjunction with Figure 
4C. One of skill in the art will immediately understand that a user can both annotate and 
bookmark the same content using the method 400. 

[0022] Figure 4B is a flow diagram for the acts performed in an annotation method 410. 
The annotations for the multimedia content selection are edited at block 411 by adding a new 
annotation, or modifying or deleting an existing annotation. In one embodiment, a user 
interface guides the user in editing the content. A template is displayed to help the user enter 
a new annotation. A list of existing annotation for the content selection is displayed to the 
user, such as in a scrollable window so the user can use the window scroll bar to visually 
search for a particular entry. When the user indicates the editing is complete, the user 
specifies how to handle the annotated content. 

[0023] If the annotated content is to be exported to an external device (block 413), the 
annotated content is stored in an external format that can be imported by other application, 
e.g., a plain ASCII file. Instead of storing the annotated content for subsequent use, the user 
may directly export the content to an external appUcation (block 417). In this case, the 
external application is automatically launched and the annotated content is automatically 
imported into the external application at block 419. The annotated content is further stored to 
an annotated content data store, such as the internal database 240 in Figure 2 (block 421). 
[0024] Figure 4C illustrates a bookmark method 430 that is performed when a user 
wishes to bookmark a content selection and search for related information. A bookmark 
identifier specified by the user is assigned to the content selection (block 43 1). The 
bookmark is typically a textual string representing a name that the user can easily remember. 
The bookmark can be assigned to any part of the content ranging from the complete content 
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to individual frames and anything in between, e.g., a scene, a screenshot. In one 
embodiment, search criteria is formulated using information retrieved from metadata 
associated with the content. The available information depends on the chosen granularity of 
the bookmark. For example, if the granularity is a scene, information regarding that 
particular scene is retrieved from the content's metadata, e.g., the name of the city where the 
scene was shot, the names of all characters shown in the scene, etc. 

[0025] If there is a network connection (block 433), the available networks are added to a 
search list (block 435). If there is an on-site media system (block 437), the available devices 
in the media system are added to the search list (block 439). The user's profile information is 
obtained to further refine the search parameters (block 441). At block 443, the method 430 
uses the search list and the user profile to find available sources containing information 
related to the content selection. The actual execution of the search depends on the entity to 
be searched. For example, to search the Internet, the search criteria may be passed to any 
standard search engine to look for URL links that satisfy the criterion. 
[0026] The results returned from the different entities are combined to form a complete 
list of related sources. The list of the available related sources is logically attached to the 
content selection (block 445). 

[0027] It will be appreciated that the content may be stored separately from the 
annotations, bookmarks, and list of related sources. It will be further appreciated that, 
although not illustrated, the augmentation manager method 400 performs periodic 
maintenance on the annotated content and the bookmarks. If particular content is no longer 
stored, the corresponding annotations will be automatically purged. Similarly, the list of 
related sources is purged of invalid sources and newly found related sources may be added. 
[0028] The annotation, bookmarking, and discovery of related information for 
multimedia content has been described. Although specific embodiments have been illustrated 
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and described herein, it will be appreciated by those of ordinary skill in the art that any 
aixangement which is calculated to achieve the same purpose may be substituted for the 
specific embodiments shown. This application is intended to cover any adaptations or 
variations of the present invention. The terminology used in this application with respect to 
networks is meant to include all networking environments, including public and private, local 
and wide area, wired and wireless. Therefore, it is manifestly intended that this invention be 
limited only by the following claims and equivalents thereof. 
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